The present invention relates to a method of coding an image sequence, comprising the steps of:
transforming the images of said sequence into successive segmented images comprising a plurality of regions and associated labels defining partitions;
estimating for the regions of the current partition a motion information about the evolution of said segmentation from the previous partition;
encoding the contour and texture of each region of said current partition;
generating on the basis of said encoded partition a corresponding reconstructed partition.
The invention also relates to a system for coding an image sequence, comprising: a segmentation device, for generating successive partitions comprising a plurality of regions, a motion estimation device, for the evaluation of the motion of said segmentation from a previous partition to the current one, an order information evaluation device, for defining a priority in the coding order of said regions, a coding device, for coding contour and texture signals for all regions of the current partition, and a partition reconstruction device. Additionally, the present invention relates to a coded signal as obtained at the output of such a system as well as to a storage medium for storing said coded signal.
The invention also relates to a method of decoding coded signals corresponding to the regions of each successive partition of a segmented image sequence, and to a decoding system for carrying out said decoding method.
In the following description, the word xe2x80x9cimagexe2x80x9d has to be understood according to its widest meaning. It is indeed clear that the invention deals not only with conventional images in which at least a luminance signal is associated to each picture element, but also more generally to any set of tridimensional data (u, v, w) among which two of them define a kind of position within a predetermined frame and the third one a value varying within a given range and respectively associated to each pair of data. The proposed method offers however a substantial interest for all segmentation-based coding techniques and can be used particularly in video terminals that use a motion compensated coding of the partition issued from the segmentation. Such techniques are mainly useful for very low bit rate applications, and a standardization under the reference MPEG4 is contemplated (MPEG is an acronym for xe2x80x9cMoving Picture Experts Groupxe2x80x9d, which is a group of experts of the International Standardization Organization ISO).
The basic idea of segmentation-based coding of image sequences is to define a partition of said images that is more appropriate for coding than the conventional block-based coding schemes. A conventional segmentation-based coding method involves mainly the following steps: definition of the partition, coding (and transmission) of the partition, coding (and transmission) of the pixel values inside each region defined by the partition. Two modes of transmission can be distinguished: the intra-frame mode, which consists in sending the information about the partition of each frame independently, and the inter-frame mode, which relies on the characterization and coding of the time evolution of the partition from one frame to the next one (and is often much more efficient than the intra-frame mode).
The inter-frame mode of partition coding generally relies on the following steps:
(a) segmentation of the frames (the present invention will address the general problem of partition coding by motion compensation and does not rely on any specific segmentation scheme);
(b) estimation of the motion of the partition between an earlier partition and the current one;
(c) prediction by motion compensation of said earlier partition;
(d) computation of the prediction error;
(e) coding of this prediction error, the coded error being then transmitted (individually for each region of the partition).
The motion estimation analyzes the movement of objects between two successive frames and leads to a motion information represented by a set of parameters that are assigned to each region. This motion information, combined with a stored reconstructed partition, allows to carry out the motion compensation, and the difference between the current segmented frame and the reconstructed partition gives the prediction error, which can be coded and transmitted.
However a distinction has to be made between the motion of the shape of a region and the motion of the pixels inside a region. Indeed, both notions coincide in the case of an object placed in the foreground of another one: either the contours of the region of the partition corresponding to that object or its texture can indifferently be considered, the pixels of the interior of this region and its contours following the same motion. This is no longer the case for a background object, because the modifications of the shape or contours of the region corresponding to this background object are generally not defined by the motion of its interior but by the motion of the foreground regions overlapping that background region.
For background regions it is therefore necessary to perform two different motion estimations, in order to have different motion compensations for contours and for textures, the first estimation being used for contour coding and the second one for texture coding.
The communication xe2x80x9cMotion and region overlapping estimation for segmentation-based video codingxe2x80x9d of M. Pardas and P. Salembier, IEEE International Conference on Image Processing, Austin, Tex. (USA), November 1994, volume II, pp.428-432, describes an object-based video coding system in which it is indeed avoided to perform two different motion estimations for the background regions. The motion estimation for contours is replaced by an extra information called order relation between regions and which allows to know for every couple of neighbouring regions which of them is in the foreground. To this end one performs for every boundary between two neighbouring regions a two hypothesis test consisting in calculating the prediction error in each region for the two possible order relations and selecting the order that yields the lowest one. It however appears that the implementation of this method of determination of said order information leads to a very high computational load: a prediction error has to be calculated twice for each region and, moreover, a lot of these computations are not useful since the corresponding regions will finally perhaps be coded according to the intra-frame mode, i.e. without re-using the error predictions.
It is therefore a first object of the invention to propose a different and more efficient coding method, leading to a reduced and less expensive computational load. This object is achieved by means of a method as defined in the preamble of the description and in which the motion information estimation step comprises in cascade the sub-steps of:
defining for each region of said current partition, on the basis of a comparison between this current partition and a partition compensated from said reconstructed partition and said motion information, an intra-frame or interframe contour transmission mode;
defining for each pair of neighbouring regions of said current partition, on the basis of a comparison between respective motion informations associated to these two regions, an auxiliary order information and an updated motion information.
The proposed method is more efficient than the one described in the previously cited document in that the determination of the order information is carried out by way of a global estimation of both the mode of transmission and the order information, said mode of transmission being taken into account for each region of the partition issued from the segmentation and no longer by considering that all regions of a given frame are transmitted in the same mode.
In a more efficient implementation, the method includes a contour transmission mode definition sub-step in which information is sent backwards in order to constitute a loop in which said order information updates the definition of said contour transmission mode.
According to the invention, this method may also be advantageously characterized in that:
(A) said contour transmission mode definition sub-step comprises for each region individually the operations of:
compensating on the basis of the motion information associated with each successive region of the current partition said corresponding reconstructed partition;
computing on the basis of a comparison between said current partition and said corresponding compensated partition a compensation error;
generating as a motion information and on the basis of a comparison between coding costs of said current partition and said compensation error a decision related to the transmission mode of the contour of the concerned region;
(B) said auxiliary order information definition sub-step comprises the operations of:
defining a motion vector for each position inside the concerned region and assigning said vector to all points of the current partition;
generating the order information corresponding to each of the concerned regions, on the basis of a comparison between said current partition and said reconstructed partition when considering said assigned vector;
quantizing said order information by comparing for each pair of labels associated with two concerned regions the number of occurrences where the first label has been declared to be in the foreground of the second one and the reciprocal number and selecting the largest one.
A further object of the invention is to propose a coding system for implementing a method as previously indicated.
This object is achieved by means of a coding system as described in the preamble of the description and in which said order information evaluation device is a decision-directed sub-system comprising a transmission mode definition circuit, provided for assessing the coding costs of the contour information in intra-frame and inter-frame modes and defining a first updated motion output signal including an updated motion information and a corresponding transmission mode information, and an order computation circuit, provided for associating to said first updated motion signal a second order signal and comprising:
motion field expansion block, in the form of a first scanning loop for extending, for each region, its motion model to all points of the partition and defining an order information for said concerned region, and of a second scanning loop for iterating said motion field expansion for all regions of each current partition;
an updating block;
a quantization block, for taking a majority rule decision related to the final order of each region;
the output of said order information evaluation device associating said final order signal and said updated motion signal.
A further object of the invention is to define a coded signal such as generated by such a coding system, said coded signal corresponding to each region of each successive partition of a segmented image sequence and comprising, for each pair of regions of a partition considered as in conflict as soon as their contour and texture motions differ, a first coded signal, corresponding to the texture signal of each concerned region and to the corresponding motion information that characterizes the evolution of said region between the previous partition and the current one, and auxiliary signals, corresponding to an order signal, defining for each considered pair of regions said in conflict which of them is in the foreground of the other one, and to an associated updated motion information. A further object of the invention is to propose a storage medium for storing said coded signal.
A further object of the invention is finally to propose a method of decoding such a coded signal, said decoding method then comprising the steps of:
(a) decoding the regions that have been previously coded in intra-frame mode and creating a corresponding mask by assigning the corresponding labels to these regions and a zero label to the other ones;
(b) generating a compensated partition made of intra-frame coded regions, motion compensated regions and holes corresponding to empty areas;
(c) decoding the error regions and generating a socalled over-partition made of the intersection between the compensated regions and the set of transmitted error regions, the labels of the regions of said over-partition being decoded and assigned; and to propose a corresponding decoding system.